Interview Q and A

This document lists frequently asked technical, architectural, and troubleshooting questions during Data Engineering interviews focusing on Amazon EMR.

Q1: What is the main difference between AWS EMR and AWS Glue? When would you choose one over the other?

Answer:

• AWS Glue is serverless (NoOps). You define the ETL code, and Glue handles scaling and executing the Spark environment on demand. Glue has quick start times (in modern versions) and is best suited for straightforward, standard Spark ETL pipelines, event-driven integrations, metadata cataloging, and quick-running jobs.
• Amazon EMR is an EC2-based managed cluster environment. You retain complete OS-level control (can SSH into nodes, tweak Spark/YARN/Hadoop core configurations, install custom packages with bootstrap scripts, and select specific GPU or high-memory instances). It is best suited for petabyte-scale data processing, long-running interactive environments (such as JupyterHub/EMR Studio), customized Spark tuning, and running non-Spark ecosystems like Apache Flink or Presto/Trino.

Q2: Why is it highly recommended to run EMR Primary Nodes on On-Demand instances instead of Spot instances?

Answer: The Primary node acts as the brain of the cluster, coordinating the metadata, scheduling, and running critical Hadoop services like the YARN ResourceManager and the HDFS NameNode. If a Spot instance is reclaimed by AWS due to capacity changes, the active Primary node will fail, terminating the entire cluster instantly and ruining any running jobs. To ensure reliability, always use On-Demand or Reserved Instances for Primary nodes.

Q3: What is the purpose of EMR Bootstrap Actions, and when do they execute?

Answer: A Bootstrap Action is a custom script (typically a shell script hosted in an S3 bucket) that EMR runs on all cluster nodes (Primary, Core, and Task nodes) during cluster setup.

• When they run: They execute before Amazon EMR installs and launches the primary applications (like Spark, Hadoop, and Hive).
• Common use cases: Installing native OS libraries (using yum), installing Python packages (using pip), setting global environment variables, or writing customized configuration files to the machines.

Q4: How does EMRFS allow you to decouple compute and storage?

Answer: EMRFS is an implementation of the Hadoop File System (HDFS) that enables EMR clusters to read and write data directly to and from Amazon S3.

• Traditional Hadoop clusters require large, expensive Core nodes containing local storage to host HDFS blocks.
• With EMRFS, your primary storage becomes Amazon S3. Your EMR cluster can read data straight from S3, run high-performance in-memory processing inside Spark, and output results directly back to S3.
• This allows you to scale down Core nodes to a absolute minimum (typically just 1 or 2 small nodes) and scale up transient, cheap compute resources (using Task nodes running on Spot instances) strictly for the duration of the processing job.

Q5: How do you prevent EMR clusters from staying active indefinitely and accumulating high costs if your Spark ETL pipeline crashes?

Answer: There are several strategies to prevent idle clusters:

1. Transient Clusters (KeepJobFlowAliveWhenNoSteps=False): When launching clusters via scripts (or AWS CLI), set this flag to False. The cluster will automatically shut down as soon as your job steps finish.
2. Orchestrator Control (all_done Trigger Rule): When using orchestrators like Apache Airflow, always map the EMR Termination task with a trigger rule like all_done or run it in a finally block. This guarantees that even if a Spark processing step fails, the termination command is called.
3. EMR Auto-termination Feature: EMR includes an idle-timeout auto-termination feature. You can configure the cluster to automatically shut down if it has been idle (no steps running, no interactive notebook sessions) for a specified number of seconds (e.g., 30 minutes).

Q6: If a Spark job running on EMR fails with OutOfMemoryError (OOM), what steps would you take to resolve this?

Answer: OOM errors can occur in the Spark Driver or Spark Executors. I would check the YARN application logs:

• Driver OOM: Usually caused by collecting too much data to the driver (e.g., calling .collect() on a giant DataFrame) or using massive broadcast joins. I would replace .collect() with writing directly to S3 or increase driver memory using --driver-memory.
• Executor OOM: Often caused by data skew or insufficient executor memory.
• Data Skew: If a few tasks are processing significantly more data than others (check Spark UI task execution times). I would repartition the data, add salt keys to the join keys, or use adaptive query execution (AQE) skew join optimization.
• Increase Resources: Adjust the Spark submission arguments (e.g., increasing --executor-memory or reducing --executor-cores to allocate more memory per core).
• Tweak Overhead: Increase YARN executor overhead memory using --conf spark.yarn.executor.memoryOverhead.

Q7: Explain the difference between Instance Groups and Instance Fleets in EMR.

Answer:

• Instance Groups: The traditional setup where you assign exactly one EC2 instance type to each group (Primary, Core, and Task). If AWS runs out of capacity for that specific instance type, your cluster creation fails.
• Instance Fleets: The modern, flexible configuration. You can specify a list of up to 30 alternative EC2 instance types for each fleet. EMR will dynamically provision the best available combination of instances based on target capacities, pricing, and live AWS Spot capacity. This is highly recommended for production Spot task nodes to maximize cluster provisioning success rates.